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A sender (101) creates a voicemail message that includes a 
plurality of messaging elements that include embedded 
instructions that define a structure of the message. Other 
messaging elements may include textual fragments, audio 
fragments that refer to audio data files that are associated 
with the message, and/or references to textual fragments 
and/or audio fragments that are stored at specified locations. 
The instructions may define a hierarchical menu structure 
which, when the message is presented to the recipient (106) 
by a messaging system (104), offers the recipient different 
navigational choices from which the recipient may select 
and thus effect the information content of what is presented. 
The structured message is sent by the sender to the recipient 
at an address on a messaging system that stores the message 
and is capable of interpreting and assembling the messaging 
elements in accordance with the embedded instruction and 
playing-out the message to the recipient when the recipient 
accesses the system and retrieves the message from storage. 
The structured message may also include links to a specific 
destination, to which the messaging system establishes a 
connection if the link is selected by the recipient during 
play-out of the message. Further, the messaging system can 
gather information from the recipient during play-out and 
forward that information to a specified destination, which 
may respond with another structured message. In the 
described embodiment, a phone markup language is used to 
define the structure and the inherent embedded instructions 
associated with the structure of the message. 
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STRUCTURED VOICEMAIL MESSAGES 

CROSS REFERENCE TO RELATED 
APPLICATIONS 

This application describes and claims subject matter that 
is also described in co-pending United States patent appli- 
cations filed simultaneously herewith and entitled: 
"METHOD AND APPARATUS FOR CREATING AND 
SENDING STRUCTURED VOICEMAIL MESSAGES", 
Ser. No. 09/318,140; and "METHOD AND APPARATUS 
FOR ASSEMBLING AND PRESENTING STRUCTURED 
VOICEMAIL MESSAGES", Ser. No. 09/318,450, nowU.S. 
Pal. No. 6,240,391, issued on May 29, 2001. 

TECHNICAL FIELD 15 
This invention relates to voice mail messaging. 

BACKGROUND OF THE INVENTION 

Messaging systems allow a message recipient to listen to 20 
an audio message via his telephone or other audio terminal 
In so-called voicemail systems, when the message is 
accessed from the voicemail system, the voicemail system 
typically presents header information, such as the time of 
receipt of the message and the identity of the sender, if 25 
known, and plays a recorded message, consisting of a 
segment of audio material, to the recipient. The recipient can 
navigate through the recorded message using his telephone 
keypad or voice input that can effect a skip, rewind, pause, 
or other similar operations. Recently, integrated messaging 30 
systems have been introduced that have voice interfaces that 
can handle conventional voicemail messages as well as 
messages of other media types, such as email. In the latter 
case, a textual email message is delivered to recipient's 
mailbox. When retrieved by the recipient through his audio 35 
terminal, the email header information is converted to audio 
and presented to the recipient together with the body of the 
message, which is played for the recipient using tcxt-lo- 
speech processing. Thus, in both the traditional voicemail 
systems and the integrated messaging systems, the body of 40 
the message is interpreted as a monolithic chunk of recorded 
audio or text, the latter being converted to audio, which 
audio in either case is played linearly to the recipient when 
he accesses his messaging system from his telephone or 
other audio terminal. 45 

SUMMERY OF THE INVENTION 

Voicemail and other messaging systems have revolution- 
ized the way people communicate with each other in today's 50 
electronic age. Although the messaging systems available 
today are generally useful and have found widespread 
popularity, we have recognized that additional and highly 
advantageous functionality can be achieved in accordance 
with our invention. S5 

The present invention is directed to a structured message 
that includes a plurality of messaging elements. The sending 
of such a message to a recipient's messaging system is the 
subject of co-pending patent application Ser. No. 09/318, 
140, filed on even date hereof. The assembly and presenta- <sn 
tion of the message, as described in further detail herein 
below, is the subject matter of patent application Ser. No. 
09/318,450, also filed on even date hereof, and now U.S. Pat. 
No. 6,240,391, issued on May 29, 2001. 

The structured message sent by a sender includes a 65 
plurality of messaging elements. These messaging elements 
may illustratively include textual fragments, speech frag- 
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ments in attached audio files, references to audio or textual 
fragments stored at specified addresses, and explicit or 
implicit instructions that define the structure of the message. 
The message, including a plurality of such messaging ele- 
ments is delivered to an address indicated in the message of 
the recipient's mailbox on a messaging system that has the 
capability of interpreting the instructions incorporated 
within the structured message. That messaging system, upon 
retrieval by the recipient, assembles, in accordance with the 
instructions that define the message structure, an audio 
message using the messaging elements associated with the 
message content, and presents that assembled message to the 
recipient in its intended format. 

Advantageously, the delivery of the structured message 
may enable interactions between the recipient and the mes- 
sage content, and between the recipient and the outside 
world. In particular, the embedded instructions within the 
message may be such as to allow a dialog between the 
recipient and the messaging system. Indeed, that dialog can, 
in accordance with the embedded instructions, allow the 
recipient to navigate between messaging elements through 
voice and/or keypad inputs, as if the recipient was connected 
to an active interactive voice response (IVR) system. The 
recipient will thus hear those content-related messaging 
elements from within the structured message that are asso- 
ciated with and are responsive to his command inputs. 

The structured message may also contain embedded 
addresses, or "links" as they are currently known in the 
Internet art, that specify a telephone address such as a 
telephone number, or an IP telephony address. If the recipi- 
ent performs an action, such as making a keypad entry or 
supplying a voice input, during his interaction with a struc- 
tured message, which action is interpreted by the messaging 
system to represent a selection by the recipient of a specific 
link, placement of a call to that telephone number or address 
associated with that link is effected by the messaging sys- 
tem. Alternatively, the structural message may contain 
embedded links that specify a destination for messaging 
rather than telephony connections. Examples of the latter 
include email addresses and Web services for HTTP upload. 

The messaging system can gather information from the 
recipient during his interaction with the structured message, 
which information is then sent to a destination specified, for 
example, by the sender, such as a server or email address. 
Receipt of that gathered information may result in a 
response from the specified destination, which response is 
processed by the messaging system and forwarded to the 
recipient. This, in effect, initiates an interactive session 
between the recipient and a service that is active at the 
destination specified in the original structured message. 

The various capabilities of the structured message can 
also be combined in several ways. As an example, a struc- 
tured message may cause coordinated data and telephony 
actions. Thus, the messaging system can collect input data 
from the recipient, communicate that data to a specified 
destination system, such as a server, and place a telephone 
call to a phone number associated with that destination 
system. That destination system can then be provided with 
information over the telephone call that enables it to access 
the separately sent data. The destination system then may 
use that data to enhance the handling of the telephone call in 
various ways. 

In a specific illustrative embodiment, the structured mes- 
sage is formulated by a sender using, for example, a phone 
markup language (PML) to define the structure and the 
inherent embedded instructions associated with the structure 
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of the message. The message then consists of PML markup 
interleaved with other messaging elements such as textual 
fragments that will be converted to speech by the messaging 
system, and/or audio and textual fragments made part of the 
message as attached files or which are retrievable from a 
designated address. After formulating the message, the com- 
posite message is sent over a data network, as for example, 
an IP network such as the Internet, to the messaging system 
which stores the composite message for later retrieval by the 
intended recipient. 

The messaging system includes those functionalities nec- 
essary to interpret the embedded instructions within the 
stored structured message and to audibly present it to the 
recipient, while also being able to receive and interpret a 
recipient's audio or touch-tone inputs for interaction with 
the message in accordance with the instructions. In the 
specific illustrative embodiment, the messaging system 
receives and stores the PML-formatted message sent by the 
sender over the data network. Upon being accessed by the 
recipient for retrieval of the message, the system accesses 
the message, and a processor interprets the PML markup 
within the message to effect playing of the textual and/or 
audio fragments of the message to the recipient in accor- 
dance with the embedded instructions associated with that 
markup. Thus, for example, for a structured PML-formatted 
message including fragments of text and attached audio 
files, the message is formulated by converting the text to 
speech using a text-to-speech processor, and inserting the 
appropriate audio file(s) during the play-out to the recipient 
in the proper sequence, as determined by the embedded 
instructions within the PML-formatted message. Further, the 
illustrative messaging system includes a detector for delect- 
ing the recipient's touch-tone keypad inputs and an auto- 
matic speech recognizer (ASR) processor for recognizing 
and interpreting the recipient's voice and touch-tone inputs 
to effect interaction and navigation within the structured 
message as allowed by the markup within the message, as 
well as the transfer to and interaction with other destinations 
as specified by the markup. 

Advantageously, the structured message can be formu- 
lated by the sender through an editor with a graphical user 
interface running on a computer. Through the input of 
textual files, previously recorded audio fragments, as well as 
contemporaneously recorded fragments, the sender is able to 
formulate the structured message. Alternatively, the struc- 
tured message could be created "by hand" with a text editor 
and an audio file recording utility. 

Advantageously, if the structured message is sent to a 
plurality of recipients, the invention allows information to 
be gathered from each, without requiring real-time tele- 
phonic communications with each individual recipient to 
collect that information. 

Although noted above as being associated with audio 
messaging, it should be understood that the present inven- 
tion could equally be applied to multi-media type of mes- 
saging in which the messaging elements of the structured 
message may include video fragments that are assembled by 
the messaging system in accordance with the instruction 
embedded within the structured message. 

BRIEF DESCRIPTION OF THE DRAWING 

FIG. 1 is a block diagram of a system incorporating a 
messaging system that is capable of receiving, storing, and 
presenting to a recipient a structured message; 

FIG. 2 is flowchart showing the steps associated with 
sending and delivering a structured message to a recipient; 
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FIG. 3 is a flowchart showing the steps associated with 
sending a customized structured message to a plurality of 
recipients; 

FIG. 4 is a flowchart associated with the steps of sending 
s a structured message to a recipient which allows interaction 
between the recipient and the message; 

FIG. 5 is a flowchart associated with the steps of sending 
to a recipient a structured message which has embedded 
links; 

FIG. 6 is a flowchart associated with the steps of sending 
lo a recipient a structured message in which, during a 
recipient's interaction, information is collected and sent to a 
specified destination; 

!5 FIG. 7 is a flowchart associated with the steps of sending 
to a recipient a structured message in which, during a 
recipient's interaction, information is retrieved from a speci- 
fied from a specified destination to enable further interaction 
between the recipient and that information; 

20 FIG. 8 is a flowchart associated with the steps of sending 
to a recipient a structured message in which, during a 
recipient's interaction, information is collected from the 
recipient and sent to a specified destination, and a separate 
call is established between the recipient and that specified 

25 destination; 

FIG. 9 is a block diagram of an integrated messaging 
system that receives, interprets and presents a structured 
message; and 

FIG. 10 is a block diagram of a graphical user interface 
30 (GUI) for creating a structured message. 

DETAILED DESCRIPTION 

With reference to FIG. 1, a system incorporating and 

35 using a structured message is shown. A sender at a client 
terminal 101 running an editor 102 with graphical user 
interface (GUI) 103 prepares the structured voiceraail 
message, which is transmitted to an integrated messaging 
system 104 over a wide or local area data network, such as 

40 an IP network 105 like the Internet or an Intranet. Integrating 
messaging system 104 is a messaging system that is capable 
of receiving conventional voicemail messages, email 
messages, and structured messages of the type described 
herein. With respect to the structured messages, messaging 

45 system 104 is capable of receiving, storing, interpreting, and 
delivering such structured messages to the intended recipient 
when he accesses his mailbox through an audio terminal, 
such as a telephone set 106 connected to the PSTN 107. The 
messaging system 104, as will be described, is also capable 

50 of accepting and processing touch-tone keypad or voice 
inputs from the recipient received during the recipient's 
interaction with the structured message. Although the recipi- 
ent is shown in FIG. 1 connected by his telephone 106 to 
messaging system 104 via PSTN 107, it is recognized that 

55 the recipient could also be connected via his telephone set to 
the messaging system over an IP-telephony connection, or 
over any other type of analog or data network. 

The structured message includes a plurality of messaging 
elements. These messaging elements illustratively include 

60 large or small textual fragments that, when formulated for 
presentation to the recipient, are converted by the messaging 
system lo speech; large or small audio or textual fragments 
contained in files that are attached as separate files to the 
structured message; large or small textual or audio frag- 

65 ments that are located and retrievable from a specified 
address on the network on which the messaging system 104 
is located; and implicit or explicit embedded instructions 
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that define the structure of the message. The latter includes 
not only the order in which the fragments are to be audibly 
presented to the recipient when the message is retrieved by 
the recipient from the messaging system 104 via telephone 
106, but also define, illustratively, the inputs that may need 
to be collected from the recipient through his keypad or 
voice input over telephone 106 during the dialog of the 
message. Further, the instructions define the messaging 
elements to be audibly presented to the recipient in response 
to such recipient inputs. Even further, these embedded 
instructions may also define the actions messaging system 
104 is to take in response to a recipient's inputs, which 
actions are separate from navigating within the message, 
such as establishing a connection to another location. The 
message further includes the recipient's address on a mes- 
saging system that is capable of interpreting the instructions 
embedded within the message. 

As will be later described, the structured message is 
prepared by the sender from the graphical user interface 103 



10 



lished Under the Patent Cooperation Treaty (PCT), Publi- 
cation Number WO 97/40611 entitled "Method and Appa- 
ratus For Information Retrieval Using Audio Interface", 
published Oct. 20, 1997 and claiming a priority dale of Apr. 
22, 1996 based on a co-pending U.S. patent application Ser. 
No. 08/635,801 to M. A. Benedikt, D. A. Ladd, J. C 
Ramming, K. G. Rehor and C. D. Tuckey; D. L. Atkins, T. 
Ball, T. R. Baran, M. A. Benedikt, K. C. Cox, D. A. Ladd, 
P. A. Malaga, C. Puchol, J. C. Ramming, K. G. Rehor, and 
C. D. Tuckey, "Integrated Web and Telephone Service 
Creation", Bell Labs Technical Journal, pp. 19035, Winter 
1997; and U.S. patent application Ser. No. 09/168,405, filed 
Oct 6, 1998 to M. K. Brown, K. G. Rehor, B. C. Schmull 
and C. D. Tuckey entitled "Web-Based Platform for Inter- 
active \bice Response (IVR)", all of which are incorporated 
by reference herein. 

As described in these aforcnotcd references, an end user 
at an audio terminal, such as a telephone, can access IVR 
services on an IP network through the telephone/IP server 
that interfaces the PSTN voice network and the IP network, 



on editor 102 from a collection of pre-recorded audio files 20 such as the Internet or other wide area or local area corn- 



containing either small or large voice fragments, through the 
real-time input of audio fragments, through the input, from 
a keyboard, of large and/or small textual message fragments, 
and/or through reference to other textual or audio files. For 
the specific embodiment disclosed herein, editor 102 for- js 
mulates the structural message using a phone markup lan- 
guage (PML) to define the textual and audio elements of the 
message and the embedded instructions within that message. 
As will be apparent from the specific examples later 
described in detail, the resultant message is a "page" con- 
taining PML markup that is similar in visual appearance to 
an HTML "page" created by an HTML editor used for 
creating conventional Web pages delivered for visual dis- 
play on a video terminal such as a television or a computer 
monitor. The PML "page" created by editor 102 that repre- 
sents the message thus contains PML markup and textual 
components that will be converted into audio by the mes- 
saging system 104. Further, the resultant structured message 
may also includes attachments that may be actual audio or 



30 



35 



textual files that are attached to the message, or addresses on 40 language. 



putcr. network. The telephone/IP server functions to enable 
end users to engage in interactive services via their tele- 
phone set with Web servers connected on such a wide area 
or local area network. The telephone/IP server, as described 
in the references, is embodied as hardware and software on 
a general purpose computer that together perform the func- 
tions of audio play and record, text-lo-speech synthesis, 
dual-tone multi-frequency (DTMF) (touch-tone) 
recognition, automatic speech recognition (ASR) 
processing, and other call control functions necessary for 
interactive audio services. Such a telephone/IP server func- 
tions to accept inputs from the telephone end user as speech 
or DTMF signals, and act as a proxy browser for that end 
user in making requests over the Internet to those Web 
servers that provide the IVR services with which the end 
user wishes to interact- The telephone/IP server and the Web 
servers providing the IVR services communicate using a 
PML. As noted, PML will be supplanted in the future with 
the expected- to-be standardized voice extensible markup 



the IP network 105, such as URLs, from which audio or 
textual files can be accessed and downloaded to the mes- 
saging system. 

A phone markup language is a language that has been 
implemented to allow users at audio terminals, such as 
telephones, to interface with Web-based voice response units 
to access interactive voice response (IVR) services and to 
access other Internet-connected Web servers. (See, e.g., 
"PML: A Language Interface to Networked Vbice Response 
Units", by J. C. Ramming, Workshop on Internet Program- 
ming Languages, 1CCL J 98, Loyola University, Chicago, 
111., May, 1988, which is incorporated by reference herein). 
Recently, AT&T, Motorola, IBM and Lucent Technologies 
announced, a joint cooperation activity for implementing a 
voice extensible markup language that will allows end users 
at voice terminals to access the Internet by voice (See, e.g., 
Wall Street Journal, Mar. 2, 1999). That language is 
expected to become a standard for defining voice commands 
to the Internet and is likely to incorporate many aspects of 
the PML described in the above-cited paper by J. C. Ram- 
ming. Such Web-based interactive voice services are pro- 
vided to telephone users through what shall hereinafter be 
called a telephone/IP server. Such a telephone/IP server 
terminates a telephone call on one side, and is connected to 
the IP network on the other. The ability for an end user at an 
audio terminal, such as a telephone, to access the Internet is 
described in, for example, International Application Pub- 



45 



50 



55 



fiO 



65 



We have recognized that advantageous results can be 
achieved by incorporating the functionalities of the 
telephone/IP server into an integrated messaging system 
104. A structured message prepared by the sender from, as 
an example, client terminal 101, and which includes a 
plurality of messaging elements that are formatted with PML 
markup, or the like, is sent to such an integrated messaging 
system 104 having these functionalities and then stored. 
When the recipient accesses messaging system 104, the 
message is retrieved from storage and processed in accor- 
dance with the embedded instructions with the PML 
marked-up stored message. The messaging system 104, thus 
includes an interpreter that is able to interpret the embedded 
instructions and audibly present the message to the recipient 
in the manner intended by sender. Thus, the various mes- 
saging elements may include a combination of textual 
fragments within the body of the message, audio and/or 
textual fragments in data files attached to and stored with the 
message, and textual and/or audio fragments stored at speci- 
fied URLs on IP network 105. For play-out, the textual 
messaging elements are converted to speech by a text-to- 
speech processor, and combined with each other and audio 
fragments, converted from their data files, so as to present an 
audio message to the recipient that, to the recipient's ears, 
has the aural appearance of a unified message. 

Advantageously, the structured message sent by the 
sender to the recipient's integrated messaging system 104, 
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may be structured with the PML markup to enable interac- ing message to everyone who signs up for a course. The 

lions between the recipient and the message content, and message is an email automatically generated by the regis- 

betwcen the recipient and the outside world, when the tration system 110 and sent to the registrant's mailbox on the 

recipient retrieves the message. The PML-formatted and integrated messaging system 104. The generated message 

stored message may thus include markup that is interpreted 5 sent may appear as shown in Table 1. 
by the interpreter to present, for example, a selection choice 

to the recipient. The interpreter, in presenting the message to TAR IF, 1 

the recipient, would then expect to receive an input from the 

recipient in response to the selection choice. The recipient's <PM ™.„ m „ DiW « D «„ m , 

S . . \ . „ . 4 . . . , 1 .J1j <AUDIO SRC-~inspiratioDal.au BACKGROUNDS 

particular input, inputted via the touch-tone keypad on <audio SRC--fouu"/> 

telephone set 106, or by voice, which an automatic speech p etcI Mataga 

recognition (ASR) processor recognizes, determines how <audio SRC-~&om.au-/>. 

and what is to be presented to the recipient within the <audio SRC--thanks.aif7><AUpto SRC--ccurK.air/> 

UM301: Advanced Unified Messaging, 

message. <AUDIO SRC-~couise.au VxAUDIO SRC--wben.au' V> 

The details of the messaging system as it receives and 15 January 26, 1999 from 3:00pm to 5:00pm. 

stores a structured message, and then processes that message <J?ML> 
when later retrieved by the recipient will be discussed 

hereinafter, as will the details associated with preparing the The message also includes six MIME attachments, audio 
structured message from the sender's end. A better under- files: "inspirational.au", "for.au", "from.au", "thanks.au", 
standing of the invention can be achieved by first examining ^ "course.au" and "when.au", which arc retrieved from within 
several specific examples of structured messages below. In the PML-formatted message. To play the message to the 
the examples described below, it is assumed that the inte- recipient when he retrieves it by accessing his mailbox on 
grated messaging system to which structured messages are messaging system 104, messaging system 104 assembles 
sent and from which messages are retrieved is located on a information from the message, using the textual part of the 
corporate intranet so that the PML used to format the ^ message in two ways. First, the markup elements, delimited 
messages can be standardized between all message senders by angle brackets in the example markup used in Table 1, are 
and the messaging system can easily be arranged. A mes- used to determine the message structure. Second, textual 
saging system that is available to receive structured mes- content occurring within or between the markup elements 
sages from any recipient would require a larger scale stan- are synthesized to speech at an appropriate point in the 
dardization of the phone markup language, which as 30 play-out of the message to the recipient. In this example, the 
previously noted is the intent of the announced joint project message structure is simple, requiring playing of audio files 
between AT&T, Motorola and Lucent Technologies. The IP and performing text-to-speech (TTS) processing in an inter- 
network 105, in the examples that follow, is thus assumed to leaved (and possibly parallel) fashion, 
be a corporate Intranet to which the sender at client terminal The recipient at telephone set 106 hears the following 
101 is also connected. 35 when this structured message is outputted by messaging 
For ease of understanding the examples that follow, a system 104. To aid in understanding the functions of the 
common scenario is postulated. Specifically, in these messaging system, the audio-file generated fragments are 
examples, the postulated scenario is that the sender of the underlined below, while the TTS fragments are shown in 
structured messages is an internal corporate training orga- plain text: 

nization that wants to inform employees of course 40 (inspirational music plays in background . . . ) 

availability, and that such employees be able to access "This is a message for Peter Mataga from the training 

information and to register for such courses through their organization. Thank you for registering for the course 

telephones. Rather than dedicate a voice response system 'UM301: Advanced Unified Messaging'. The course 

and to support the telecommunications infrastructure (e.g., w m uke place Jan. 26, 199 from 3:00 pm to 5:00 pm." 

many incoming lines) needed for such purposes, structured 45 The text fragments "Peter Mataga", "UM301: Advanced 

messages are sent to the mailboxes of employees on an Unified Messaging", and "Jan. 26, 1999 from 3:00 Pm to 

integrated messaging system 104, which is capable of under- s.00 pm" are converted to speech. The background music 

standing and interpreting such structured messages. A par- an d the audio fragments "This is a message for", "from the 

ticular PML is assumed herein as the structuring mechanism training organization", "Thank you for registering for", "the 

in the examples that follow, with the interpretation of the 50 course", and "will take place" are generated by playing the 

markup described. audio files of appropriate names (identified in the markup), 

The general structured message of the present invention after detaching them from the structured message, 

mixes a plurality of textual fragments and audio fragments The flowchart in FIG. 2 illustrates the steps of the method 

at a fine granularity. This capability is likely to be most of sending and receiving a basic structured message of the 

useful for automatically generated messages with fixed 55 type just described. At step 201, the sender sends the 

recorded audio, and generated text content. The mechanism structured message to an integrated messaging system 

for fragment assembly is through the use of embedded capable of receiving and interpreting the embedded instruc- 

markers that indicate when an audio file and when text-lo- lions within the message and presenting the message to the 

speech processing needs to be performed on a textual recipient in accordance with those instructions when the 

fragment. The audio file, or a textual file, may be part of the 60 recipient accesses the mailbox. At step 202, the messaging 

message itself (e.g., an email attachment, or multipart system stores the message sent by the sender. At some later 

MIME message), or can be retrievable from elsewhere in the time, at step 203, the recipient accesses the messaging 

network 105 to which the messaging system is connected via system and retrieves the stored structured message. At step 

a URL. Audio content may be sequential with other content 205, the messaging system interprets the structure of the 

in the structured message, or played in the background. 55 retrieved message from the embedded instructions and for- 

As a first example of the structured message, it is postu- mulates the message from a combination of: TTS processing 

lated that the training organization wants to send a confirm- of text fragments in the message body, text fragments in files 
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attached to the message, and/or text fragments at designated are combined to formulate the message to be sent to the 
URLs; and/or playing audio fragments in attached data files particular recipient's mailbox. The recipient-specific mes- 
or from addresses at designated URLs. At step 206, the saging element can be incorporated as text within the 
messaging system audibly presents the formulated struc- structured message, or can affixed to the message as an 
lured message to the recipient. After listening to the s attached textual or audio data file. At step 304, the combined 
message, at step 205, the recipient may delete, or replay the structural message is sent to the intended recipient's mail- 
message, retrieve another message, structured or not, from box At step 305) a determination is made whether any other 
the messaging system, or exit the messaging system. recipients are to receive the structured message. If yes, the 

Whereas a message sender can individually specify a flow retums t0 step 302 t0 one or more 

structured message of this type, such a message can also be 10 rehted m m elements mal are ^th another 

automatically generated. The ability to automatically gen- ^ ^ fc which ^ thcQ Qomhiacd ^ thc non . 

erate a structured message is of particular use when a sender chmcn{£ {q {oim ^ {ft a m ^ 

such as the training organization m this example, must send u ■ u- • ■ -n. 

out many such meLges to each of the individuals who have sa & e mat 15 ^ 06X1 recl P iem . s ma *°*- 

registered for a course. Thus, for this case of automatic is ^ structurcd mcss *g c ^ cont " n a substructure that 

message generation, the audio files would most likely rep- ^ red P ient t0 navi S ate lhe messa & e wlthout 

resent information that is the same for all the messages of Uneari y listening to all of the message content that was sent 

this type, while the embedded text would be dynamic b Y thc residcs in tnc recipient's mailbox. An 

information that changes from individual message to indi- example is that of a message divided into sections with 

vidual message, which information can be retrieved, for 20 headings for which the messaging system might read all the 

example, from a database 111 or a transaction system. headings and then allow the recipient to choose one or more 

FIG. 3 is a flowchart showing the steps associated with sections to listen to. Sections might also have subsections, 

formulating and sending a customized structured message to and so forth. As an example, the training organization sends 

a plurality of recipients. At step 301 the sender formulates a out a weekly informational email describing the current 

structured message that includes as content-related messag- 25 course offerings to its employees' mailboxes. A recipient is 

ing elements only those messaging elements that are com- not likely to want to listen to all of the information in the 

mon to the message sent to each recipient's mailbox. At step message, but only to those parts that are of particular interest 

302, one or more content-related messaging elements that to him. Table 2 is an example of such a message. It should 

are associated with a particular recipient are accessed from be noted that in this example, and the examples that follow, 

another source, such as a database or a transaction system. 30 for readability purposes, textual content rather than audio 

These one or more messaging elements can be textual or files are generally used. However, embedded audio content, 

audio in nature. At step 303, thc recipient-specific messaging as was used in the example in Table 1, could be substituted 

elements and the non -recipient- specific messaging elements anywhere l ib processing is employed. 



TABLE 2 



<PML> 

This is your monthly training course update. 
New courses in the unified messaging area are available this week, 
aa well as our existing course offerings. 

<MENU> 

<PROMPT></ PROMPT> 
<n"EM><TITLE>New COurs es <TTTLE> 

There are several new courses available this week. 
<MENU> 

<PROMPT> Please select a course to hear a brief 

description: </FROMFT> 
<rrEM>^TTTLE>Iiitroduaory Unified Messaging</TTTLE> 
This course covers ... 

</ITEM> 

<ITEM>^TrTLE>Advanced Unified Messaging</nTLE> 

This course covers ... 
</ITEM> 
</MENU> 
</TTEM> 

<rreM><TfTT_E> Existing Courses </TTTlE> 

We provide courses in a variety of categories. 
<MENU> 

<PROMPT>Please select a course category :<rt>ROMPT> 
<riT.M> <TTTLE> Widget Course* </TTTUi> 
<MENU> 

<PROMPT> Select a course to hear a brief 

description: </PROMPT> 
<lTEM><TrrLE>Introductory Widgets </Tm£> 
This course covers ... 

</rrE.M> 

<rTEM><TiTLE>Iatennediate Widgeta</nTLE> 

This course covers ... 
</ITE.M> 

<ITEM><TrrLE>Advanced Widgets <^nTLE> 

This course covers ... 
</ITEM> 
</MENU> 
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TABLE 2 -continued 



<ITEM>cnTLE>Gadgct Courses -/TITLE > 
<MENU> 

<PROMPT> Select a course to hear a brief 

description: </PROMFT> 
<ITEM>cTrrLE>Iiitroductory Gadgets </TlTLE> 

This course covets ... 
</ITEM> 

<rrEM><nTLE>Iiitennediate Gadgets </nTLE> 

This course covers ... 
</ITEM> 

<ITEM><TrrLE>Advanced Gadgets </TTTLE> 

This couise coves ... 
</ITEM> 
</MENU> 
</TTEM> 
<fMEXU> 
</TTEM> 

<ITEM > <nTLE> Ge nera] Infonnation</TTrLE> 

General inquiries about training should be directed to 
our main oEcc at area code 630, 979-1960. 
</TTEM> 
</MENU> 

Goodbye. We hope to see you in one of our courses toonl 
</PML> 



When the message in Tabic 2 is received by the messaging 25 
system 104, stored, and later retrieved by the recipient, the 
interpreter recognizes the markup within the message and 
from that the structure of the message, thereby enabling it to 
interpret that structure for presentation to the recipient, 
where that structure in this example defines a set of hierar- 30 
chical menus. In the example in Table 2, unlike the example 
in Table 1, the markup elements have content, which may 
also be structured. For example, a MENU element begins 
with the markup <MENU> and ends with the markup 
</MENU>. In between may occur a PROMPT element 35 
(possibly with no content) and one or more ITEM elements, 
each of which must enclose a TITLE element followed by 
any kind of content, including another MENU. 

The MENU element represents an opportunity for the 
recipient while listening to the message to make a naviga- AQ 
tional choice. The integrated messaging system processes a 
MENU element, when encountered during processing and 
play-out, by playing to the recipient the content of the 
PROMPT, which can be any combination of audio files and 
TTS information, and offering the TITLES of the ITEMS as 
choices. It should be noted that only ITEMS that occur 45 
immediately as content of the MENU element arc treated in 
this manner — ITEMS that occur deeper in the hierarchy, as 
content of enclosed MENU elements, are not used until the 
MENU in which they occur is processed. The recipient 
makes a choice, by keypad entry, speech, or other signaling 50 
method, and the content of the ITEM is read to the recipient, 
possibly involving other complex processing. The default 
behavior assumed herein for menus in this example is that 
the item is processed, then the menu is presented again. The 
recipient may choose to quit from a menu, in which case 55 
processing of the message continues after the menu. If a 
menu is part of an item in a parent menu, reprompting for the 
parent menu will result. The quit choice presented to the 
recipient might also be sensitive to whether the menu is a 
top-level menu or not. It should be noted, there arc many 60 
other possible behavior custom izations for menus; the 
markup could include instructions to the messaging system 
that control such behavior. The example presented in Table 
2 is thus only one example of a possible menu customiza- 
tion. 65 

The effect of the interpretation of the message structure by 
the messaging system is that the recipient can navigate 



through the information in the message, listening (possibly 
more than once) only to the parts in which he is interested. 
A possible recipient interaction scenario for the received 
message in Table 2 is presented below. In the scenario below, 
italicized phrases indicate audio boilerplate that is generated 
by the interpreter 117 in the messaging system 104 to present 
menus and other standard interactions, while plain text is 
TTS translations of fragments from the message, such as the 
item titles; 

parenthetical items denote the actions taken by the recipient: 
"This is your monthly training course update. 
New courses in the unified messaging area are available 

this week, as well as our existing course offerings. 
For New Courses, press 1; 
For Existing Courses, press 2; 
For General Information, press 3; 
To exit this menu, press pound sign." 
(Recipient presses 2) 

"We provide courses in a variety of categories. 

Please select a course category: 

For Widget Courses, press 1; 

For Gadget Courses, press 2; 

To return to the previous menu, press pound sign." 

(Recipient presses 1) 

"Please select a course to hear a brief description: 

For Introductory Widgets, press 1; 

For Intermediate Widgets, press 2; 

For Advanced Widgets, press 3; 

To return to the previous menu, press pound sign." 

(Recipient presses 3) 

"This course covers . . . 

For Introductory Widgets, press 1; 

For Intermediate Widgets, press 2; 

For Advanced Widgets, press 3; 

To return to the previous menu, press pound sign." 

(Recipient presses #) 

"Please select a course category: 

For Widget Courses, press 1; 
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For Gadget Courses, press 2; 

To return to the previous menu, press pound sign." 

(Recipient presses #) 

"For New Courses, press 1; 

For Existing Courses, press 2; 

For General Information, press 3; 

To exit this menu, press pound sign." 

(Recipient presses #) 

"Goodbye. We hope to see you in one of our courses 
soon!" 

(At this point the recipient receives the usual messaging 
system options to delete the message, listen to the message 
again, etc.) 

Processing of this kind of structure still involves extrac- 
tion of fragments from the message in Table 2 to present to 
the recipient, as in the message in the example of Table 1. 
However, the extraction process is more complex, requiring 
finding all item titles for a menu, constructing audio boil- 
erplate to indicate the choices, and significantly, depends 
dynamically on interactions with the recipient, which deter- 
mine specifically those portions of the structured message 
that are actually audibly presented to the recipient. 

The flowchart in FIG. 4 illustrates the steps associated 
with sending to a recipient a structured message which 
allows interaction between the recipient and the message, as 
described above. Steps 401, 402 and 403 are the same as 
steps 201, 202 and 203 in the flowchart in FIG. 2, described 
above, and for brevity, are not repeated herein. At step 404, 
the messaging system interprets the structured message, 
audibly offering navigational choices to the recipient, ,the 
navigational choices and the presentation of such choices 
being determined in accordance with the embedded instruc- 
tions within the message. At step 405, the recipient, when 
presented with a navigational choice, selects an option, as an 
example, via keypad or voice input through his telephone 
set. At step 406, the messaging system, in response to and as 
a function of the recipient's input, continues to deliver audio 
content to the recipient from within the structured message. 
At step 407, a determination is made whether the message 
is completed. If yes, at step 408, the recipient can delete or 
replay the message, retrieve another message, or end his 
interaction with the messaging system. If, at step 407, the 
message is not complete, the presentation of the structured 
message to the recipient continues at an entry point follow- 
ing the previous navigational choice, further navigational 
choices being offered to the recipient if the structured 
message is so formatted. 

The structured message may also contain embedded 
addresses, or "links" as they are currently known in the 
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Internet art, that specify a telephone address such as a 
telephone number (e.g., 630-555-5555 of a telephone set 115 
on PSTN 107), or an IP telephony address (e.g. 
sip :krehor@ipt. training at client terminal 116). The link may 

5 optionally also contain additional information, sucb as 
DTMF information, which can typically represent an 
"extension" specified as a string of keypad numbers, symbol 
and/or letters. In the packet telephony case, the link may 
specify other kinds of additional information, to be passed 

10 along with the call setup message. For example, a SIP 
INVITE could include additional header information as 
specified in appropriate link attributes. 
To process such a link the messaging system 104 offers 

is the recipient, for example, an opportunity to make an 
outgoing call to that linked address. If the recipient performs 
a responsive action to that offer, such as making a keypad 
entry or supplying a voice input, which command is inter- 
preted by the messaging system to represent an affirmative 

20 election by the recipient to select that link, the messaging 
system 104 dials out to that telephone number over PSTN 
107 (or the equivalent for non-PSTN destinations) and, on 
answer, plays the sequence of DTMF tones corresponding to 
the "extension", and then allows the recipient to continue on 

IS telephone set 106 with the just established call to the 
answering party at telephone set 115. It should be noted that 
the dialed tones could alternatively be used for some other 
general call handling purpose rather than being interpreted 
as an extension. After the recipient terminates his conver- 
ge sation over the outcall placed by the messaging system 104 
to telephone set 115, the messaging system may resume 
control for continued interaction with the recipient, thereby 
allowing the recipient to continue accessing a next message 
from his mailbox. Alternatively, the messaging may relin- 

35 quish control of the call at the time of the outcall, thereby 
effectively transferring the recipient to that call to telephone 
set 115, and exiting the session with the messaging system 
104. 

In addition to links to telephone numbers or IP telephony 
40 addresses, the message may contain embedded links that 
specify destinations for messaging rather than telephony 
connections. Examples of the these include email addresses 
and Web services for HTTP upload. If the recipient chooses 
to act on one of these links, a voice message may be 
45 recorded and sent to the specified link address as, for 
example, an email attachment. 

An example of a structured message with several types of 
embedded links is shown below in Table 3. For this example, 
it is assumed that the training organization sends this email 
to each employee's integrated mailbox. 



TABLE 3 



This is your moathly tndciog course update. 

New courses in the unified messaging area are available this week. 

Fci information, call 

<LINX TYPE-CALL DEST-"sip:krehor@ipt.traiiiiflg"> 

Ken Pehor 
</USK>. 

Foi information on existing courses, call 

<LINK TYPECALL DEST-"pstn:630.555-5555" DTMF-"3,2"> 

the automated course information system 
</UNTO>. 

As usual, general inquiries about training can be directed to 
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TABLE 3 -continued 



<UNK TYPE-MESSAGE DEST-- t mBiJto3guicy@inail3crv.iraining"> 

Sieve Gurcy 
</UNK>. 
</PML> 



This message is presented to the recipient in the usual 
sequential manner, except that embedded links are treated 10 
specially. Specifically, the content of the LINK element 
(extracted from between the <LINK> and <\LINK>markup) 
is presented to the recipient, together with additional infor- 
mation about the potential outcall. The recipient may then 
choose to have the messaging system place the call or send 15 
a message to the specified destination. 

For this example, the interaction between the recipient 
and the messaging system may be as follows. Again, boil- 
erplate audio generated by the messaging system is itali- 
cized. It can be noted that this boilerplate usually surrounds 20 
TTS fragments taken from the content and attributes of the 
markup elements, such as the description and the phone 
number of the destination. Moreover, the attributes of the 
markup elements influence the recipient's interaction. Thus, 
a LINK type of CALL is presented differently, and leads to 25 
different actions by the messaging system, from a LINK type 
of MESSAGE. It is assumed in the following example that 
the integrated messaging system 104 has automatic speech 
recognition capabilities. 

"This message contains active links. Say 'call' or 'send* to 

follow a link." 
"This is your monthly training course update. 
New courses in the unified messaging area are available 

this week. 35 
For information, call Ken Rehor at SIP address krehor at 

ipt dot training." 
(Note that the UTS text "Ken Rehor" comes from the 
content of the LINK element, while the SIP IP tele- 
phony address "krehor@ipt. training 1 ' comes from the 40 
DEST attribute of the element, which is recognized as 
a SIP URL.) 

(The messaging system pauses briefly to allow the recipi- 
ent to follow the link [i.e., make an IP telephony call 
through the network 105 to the address indicated]. If no 
action is taken, the messaging system continues.) 

"For information on existing courses, call the automated 
course information system at phone 630-555-5555, 
extension 3,2." 50 

(Messaging system pauses, recipient says 'call') "Placing 
call . . . press pound sign to terminate the call." 

(Messaging system 104 dials telephone set 115 at 630- 
555-5555, waits for answer, sends DTMF tone 3, 
pauses, then sends tone 2, conferences in recipient, then 55 
waits for call termination) 

(Recipient has conversation with called party, presses # to 
terminate outcall.) 

(Messaging system 104 disconnects outcall connection, &Q 
resumes message session) 

(Recipient chooses to hear message again; this time does 
not follow either phone link) 

As usual, general inquiries about training can be directed 65 
to Steve Gurey at email sgurey at mailserv dot train- 
ing." 



(Messaging system pauses, recipient says 'send') 
"Record an audio message at the tone." 
(Recipient records an audio message) 
"Sending message." 

(Messaging system sends the message as a MIME attach- 
ment to an email message to sgurey@mailserv.training 
on IMS 104) 

(Messaging system resumes email session) 

(Recipient chooses to hear message again; this time docs 
not follow any link. After the message has been 
presented, the messaging system offers the recipient the 
opportunity to review the links in the message, pre- 
senting the content of each LINK clement encountered 
within the message) 

"To list all the links in this messages, say 'links'." 

(Recipient says * links") 

"Call Ken Rehor;" 

(pause) 

"Call the automated " 

(Recipient barges in to say 'Rehor') 

(Messaging system places call to SIP address 
krehoraipt@training) 

A common use for the above-described facility would be 
in combination with the navigational facility of the example 
associated with Table 2, to provide a small directory service 
contained in a message. 

The flowchart in FIG. 5 illustrates the steps associated 
with sending to a recipient a structured message, which has 
embedded links as described above. Steps 501, 502 and 503 
are the same as steps 201, 202 and 203 in the flowchart in 
FIG. 2, described above, and for brevity, are not repealed 
herein. At step 504, the messaging system interprets the 
structure of the message from its embedded instructions and 
plays a message to the recipient that contains a link to a 
specified destination. At step 505, in response to presenting 
to recipient, during message play-out, a messaging element 
having an associated link, the messaging system determines 
whether the recipient responds affirmatively to accept a 
transfer to that linked destination. If, at step 505, the link 
transfer is not accepted by the recipient, at step 507, message 
play-out continues. If during the continued play-out of the 
message another link is encountered and presented at step 
508, the flow then returns to step 505 to determine whether 
the recipient accepts that link. If, during the continued 
play-out of the message, another link is not encountered at 
step 508, message play-out continues until the end of the 
message at step 509. Following the end of the message, at 
step 506, the recipient is given the option to delete, replay, 
maintain the message in storage, or exit the messaging 
system. If, at step 505, the recipient accepts a link presented 
during message play-out, at step 515, a determination is 
made of the link TYPE. If, at step 516, TYPE-CALL, then 
at step 517, a determination is made from the parameter 
DEST whether the CALL destination is a PSTN phone 
number or an IP telephony address. If it is determined to be 
a PSTN phone number, at step 510, an outcall is initiated to 
that telephone number by the messaging system. If it is 
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determined to be an IP telephony address, at step 513, the 
messaging system places an outcall to that IP telephony 
address. Following cither steps 510 or 513, if the outcall to 
the PSTN telephone number or to the IP telephony address 
docs not transfer the recipient from the messaging system to 
the outcall destination for a direct connection, then the 
recipient at the completion of the outcall, returns to the 
messaging system for continued message play-out at step 
507. If, following step 515, it is determined at step 518 that 
TYPE=MESSAGE, then, at step 511, the messaging system 
receives recipient's voicemail message and, at step 512, that 
inputted voicemail message is sent by the messaging system 



10 



18 



sends the gathered information as a data message, for 
example, to a destination specified by the sender. This 
destination could typically be a network server, such as a 
Web server with CGI, but could also be a less real-time 
destination, such as, for example, an email address. 

Table 4 is an illustrative structured message that provides 
such functionality. In this example, the training organization 
might wish to have the recipient register for a course as well 
as hearing about it. A registration request provides the 
recipient's ID as well as the course number. 



TABLE 4 



<PML> 

<FORM ACTION-"httpV/www.traiiiing/registei.cgi"> 
Thanks fcr inquiring about our training courses. 
You may register for one of our courses by responding to this 

message. 

Please provide your badge aumbei now. 

<INPUT NAME-badgenum TYPE-NUMBER MAXLENOTH-7> 

<MENU> 

<PROMPT>Pleasc select a course category: </FROMPT> 
<rreM><TTTLE>Uritfied Messaging Courses <ynTLE> 
<S ELECT NAME-courseid MULTIPLE* 

<PROMPT> Please select one or more courses: c/PROMPT> 
<OPTION VALUE-"UM1 01 ^Introductory Unified Messaging </OPTTON> 
<OPTTON VALUE-"UM301"> Advanced Unified Messaging </OPTTON*> 
</SELECT> 
</ITEM> 

<ITEM> <TtTLE>Widget Courses </nTLE> 
<S ELECT NAME-courseid MULTIPLE* 

<PROMPT> Please select one or more courses: </PROMPT> 
<OPTTON VALUE-"W10r> Introductory Widgets</OPTION> 
<OPTtON VALUE-"W20r> Intermediate WidgeU</OPTTON» 
<OPTTON VALUE-"W30r> Advanced Widgett^OPTION> 
</SELECT> 
</ITEM> 

<ITEM><TTrLE>Gadget Courses VnTLE> 
<S ELECT NAME-courseid MULTIPLE> 

<PROMPT> Please select one or more courses: </PROMPT> 
<OPTTON VALUE-"G101"> Introductory Gadgets<JOPTION> 
<OPTTON VALUE-"G2Q1"> Intermediate Gadgets <tfOPTION> 
<OPTTON VALUE-"G301"> Advanced Gadgets </OPTtON> 
</SELECT> 
</TTEM> 
</MENU> 
Thank you. 
</FDRM> 
</PML> 



to the linked email address as an attached voice data file. 
Following the sending of the recipient's voicemail message 
to the email address, the recipient is relumed to the mes- 
saging system at step 507 for continued play-out of the 
stored message. As before, during the continued message 
play-out, another link may be encountered at step 508 or 
message play-out may continue, at step 509, until the end of 
the message. At next step 506 then, the recipient can delete 
the message, replay the message, retrieve another message, 
or exit the messaging system. 

A structured message may also contain embedded instruc- 
tions that cause the messaging system to gather information 
from the recipient as, for example, by playing a prompt lhai 
requests an input from the recipient, and collecting a string 
of keypad inputted touch-tone digits, or audio-inputted 
information. The structured message thus acts as an inter- 
active form that messaging system 104 assists the recipient 
in filling out. Combination of the form inputs with a navi- 
gational structure means that the recipient may only fill out 
part of the form, if there is more than one input. The 
messaging system gathers one or more such inputs, theo 



When this message is retrieved and played to the 
recipient, the recipient is again able to navigate within (he 
message, but is this time also prompted for input at certain 
50 points, as dictated by the INPUT and SELECT markup 
elements. The INPUT element causes the messaging system 
to collect inputs from the recipient in the form of keypad- 
entered DTMF touch-tone digits or spoken characters, for 
example. The SELECT element is processed by forming a 
S5 set of choices in a manner analogous to the MENU element, 
using the enclosed OPTION elements to determine not only 
the presented choices, but the values to be returned with the 
form. The message in Table 4 might produce the following 
recipient interaction: 
6Q "This message contains a fill-out form. You may be 
prompted for information that will be senl to a service 
to process the form." 
"You may register for one of our courses by responding to 
this message. 
65 Please provide your badge number now." 

(The messaging system waits for the recipient to input 
information) 
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(Recipient presses '0123456' on his keypad. The messag- recipient has made an audio input, the ASR processor within 

ing system accepts this, without wailing for timeout or the messaging system converts the recipient's input to 

a termination symbol, because it is a con-ecUy formed textual or numerical information. At step 607, a detennina- 

numbcr exactly seven digits long, as specified by the lion is made whether the message has been completely 

TYPE and MAXLENGTH attributes of the INPUT 5 delivered to the recipient. If not, at step 608, the messaging 

element) system continues playing the message to the recipient. At 

"Please select a course category. sle P *"^» a det ennination is made whether additional input 

c" 1 1 -c j ti _ i from the recipient is requested during the continued play- 

For Unified Messaging Courses, press 1; . .. H _ \ * , . „ 

out. If yes, then flow returns to step 605. If not, the flow 

For Widget - - - 10 rcUlrns (q decision step 607. If the message has been 

(Recipient barges in with '2') completely delivered to the recipient at step 607, at step 610, 

"Please select one or more courses:" the messaging system sends all the information collected 

"Press one after each item you would like to select, or zero &ora the recipient to a destination specified in the message. 

to deselect- At ste P me recipient is then given the option to delete, 

Introductory Widgets " 15 replay ' get another mtssa Z e > or exil lhe messaging system. 

(Merging systen, pauses, recipient does nothing) u4"tf ST^ 

"To select this item, press one; to deselect this item, press othcr rccipientS) sllch ^ rcc ipients at telephone sets 121- 

zero; to clear all selections and start the list again, press X -\2\-4, from the point of view of the message sender, is the 

star; to accept current selections, press pound sign." M distribution of p roccss i ng . I n particular, with such an 

"Introductory Widgets;" arrangement it is no longer necessary to maintain a special- 

(Rccipicnt presses 1) ized service and a telecommunications infrastructure to 

"Intermediate Widgets;'* allow a plurality of sucn recipients to simultaneously call 

/r. ■, «\ into the service. 

(Recipient presses 1, then ff) , , . 

( , . „ " A structured message may also contain embedded mstruc- 

You selected two items. {iaas ^ causg ^ messaging system 104 not only t0 submit 

"Please select a course category. a fornij but to process me reS ponse from the server to which 

For Unified Messaging Courses ..." t be f orm was submitted. This response may be another 

(Recipient barges in with #) message, which the messaging system 104 processes imme- 

"Thank you." 30 di atelv - since mis new message may also be a form, the 

"-r u ,u ' f ,■ „. r, t ~„A „„„ i original message has had the effect of initialing an inlerac- 

To submit the information you entered, press 1 now. ? & 

3 r tivc session with a service active at the remote destination 

(Recipient presses 1) specified by the message sender. From the recipient's 

At this point the messaging system packages up the in for- standpoint, the perception is that of a complex and dynamic 

mation submitted by the recipient into a data message, in this 35 voice iQteractioo with the original message. 

case an HTTP request such as: In ^ ^ interactive u is also possible that no 

Grnyrcgistcr.cgi?badgcnum-oi23456&couiseid-wioi(a information at all is gathered from the user, but that the only 

courseid-waoi purpose of the action is to process the response from the 

. ...... .n-rri _ .„ remote service. This allows a message to point to informa- 

and sends it to the HTTP server 120 named www.trammg, *Q ..... . . , . . , & „ 

. . t . ... ... . ,if„ hon that is maintamed on a remote server. For example, the 

and then continues with the mailbox session, retrieving the . ... „• u . . .u - • .„ k. 

j t . ... . . , k 'j„ or ,,. m ^i... sender of the ongmal message might want the recipient to be 

next message. It should be noted that the badgenum value * . , * . , ,. r 

........ . • a. ♦ • r u * ~a k., presented with the most up-to-date version of time-varying 

submitted in the request ts the stnng of numbers entered by f . . , ^ -V; 

.u . ,u iMDirr i . ,k i .ko „™i„<.c. information. Moreover, the message size can be consider- 

the user at the INPUT element, while the courseid values n , , 7 i-„-.i„ „„„„.„i k„ 

submitted in the request arc those specified as VALUE «s f y sn^ler sm« only uie »m n. exp! c Uy req U es.ed by 

attributes in .be oniON elements corresponding to the ~P«« "« eventually be loaded mto the messagtng 

choices made by the user. sys em. 

The HTTP request would result in the information being In both cascs thc messaging system 104 is acting like a 

processed by the training organization's Web server 120. Web browser whose user, in this case the message recipient 

The HTTP response would, in this example, not be used by 50 at telephone set 106, is accessing a service on a remote Web 

the messaging system, except possibly to report an error. serv « 120 ^ ori g inaI messa S e thus acts Uke a " front 

Processing on the server 120 may lead, however, to a page" for the service. The messaging system thus may 

separate confirmation message being sent to the recipient's provide some audio cue that the presented content is in fact 

mailbox a link, such as beeping before presenting the link descrip- 

The flowchart in FIG. 6 illustrates the steps associated 55 tion. 

with sending to a recipient a structured message in which, As an example of the aforedescribed situation, and con- 

during the recipient's interaction, information is collected tinuing with the training organization example, it is assumed 

from thc recipient and sent to a destination that is specified that the training organization might send a structured mes- 

within the message. Steps 601, 602 and 603 are identical 10 sage that points to a network service that provides the 

steps 201, 202 and 203 described above, and for brevity, are 60 current offerings rather than sending a structured message 

not repeated herein. At step 604, the messaging system with all the current offerings to the mailboxes of each 

interprets the structure from the embedded instructions and recipient. This insures that the message content does not 

plays-out the message to the recipient, requesting an input become obsolete. The service might allow registration for 

from the recipient at one or more points during the message courses, and provide an immediate response to the registra- 

presentation. At step 605, when input from the recipient is 65 tion request. Table 5 is an example of a structured message 

requested during play-out of the message, the recipient that contains links that retrieve pages from the (raining Web 

makes an audio or telephone keypad entry. At step 606, if the server 120 at URL www. training. 
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TABLE 5 

<PML> 

This is your monthly trainbg course update. s 
New courses in ihe unified messaging area are available this week. 
You can get information on 

<UNK TYPE-PAGE DEST-"hUp-V/www.training/Eew^Ttr> 

new couises 
</LlNK>, 
or in far ma lion on 

<LINK TYPE-PAGE DEST-"bt^7A«ww.training/exisang.pml"> 

existing courses 
</LINK>. 
You can also 

<UNK TYPE-PAGE DEST-"http://www.trainingAegisUr.pml"> 15 

register for a course 
</UNK>, 
or get 
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(Messaging system pauses briefly) 
"or information about (beep) existing courses." 
(Messaging system pauses briefly) 
"You can also (beep) register for a course" 
(Messaging system pauses briefly) 
(Recipient says 'go') 
At this point, the messaging system retrieves the page (i.e., 
another message) from the Web server at the URL specified 
in Ihe attributes of the LINK element that the user has 
chosen to-activate, in this case sending an HTTP request: 

GET/register.pml 

to the HTTP server at URL www.training (presumed to be 
owned by the original message sender). The response to this 
request will be another page. Table 6 is an example of such 
a responsive page consisting of another structured message. 



TABLE 6 



<PML> 

<FORM ACnON""http^/www.trainiiig/register.cgi"> 

< INPUT TYPE-HIDDEN NAME-sessioaid VALUE-"98765"> 
Welcome to the course registration service. 
Please provide your badge number now. 

< INPUT NAME-badgenum TYPE-NUMBER MAXLENQTH-7> 

<MENU> 

<PROMPT>Please select a course category :</PROMPT> 
<ITEM><TrrLE>Uaifi£d Messaging Courses </TiTLE> 
<MENU> 

<PROMPT> Please select a course: </PROMPT> 
<rTTiM><TiTLE>Introductory Unified Messaging</I'l l LE> 

You selected Introductory Unified Messaging. 

•eSUBMTT NAME-course VALUE-"UM101" CONFIRM> 
</(TEM> 

< ITEM ><TTTLE> Advanced Unified Messaging VTiTLE> 
You selected Advanced Unified Messaging. 
<SUBMIT NAME-course VALUE-"UM3Q1" CONFIRM> 
</TTEM> 
</MENU> 
</ITEM> 

<rrEM><TTTLE> Widget Courses </TTTLE> 

Sorry, widget courses are oniy offered during widget hunting 

season. 

</TTEM> 

<rrEM>cTTTLE>Gadgct Courscs</TiTLE> 

Sorry, gadget courses arc no longer offered. 
</ITEM> 
</MENU> 
</FORM> 
</PML> 



TABLE 5-continucd 



<LINK TYPE-PAGE DEST- u bup:/Awww.training/gencral.pml"> 

general information about training 
</UNK>. 
</PML> 



For this example, the destination of the link identifies the 
URL of the page of markup to retrieved by the messaging 
system from the network. An example of a scenario might be 
as follows: 

"This message contains links, indicated by a beep. Say 

'go' to follow a link." 
"This is your monthly training course update. 
New courses in the unified messaging area are available 

this week. 

You can get information on (beep) new courses" 



50 with the following interaction: 

"Welcome to the course registration service. 
Please provide your badge number now." 
55 (Messaging system pauses) 

(Recipient inputs '0123456' from the keypad) 
"Please select a course category: 
For Unified Messaging Courses, press 1; 
For Widget Courses, press 2; 
For Gadget Courses, press 3. 
To exit this menu, press pound sign." 
(Recipient presses 1) 
"Please select a course: 
For Introductory Unified Messaging, press 1; 



60 



65 



12/23/2003, EAST Version: 1.4.1 



US 6,4: 

23 

For Advanced Unified Messaging, press 2; 

To return to the previous menu, press pound sign." 

(Recipient presses 1) 

"You selected Introductory Unified Messaging. 

Press 1 to continue, pound sign to cancel." 

(Recipient presses #) 

"Please select a course: 

For Introductory Unified Messaging, press 1; 

For Advanced Unified Messaging, press 2; 

To return to the previous menu, press pound sign." 

(Recipient presses 2) 

"You selected Advanced Unified Messaging. 

Press 1 to continue, pound sign to cancel." 

(Recipient presses 1) 
At this point, the messaging system submits another form, 
sending an HTTP request 

G ET7 regis tcr.cgi?eessionid-98'765 fcbadge mitn-01 23456& 
couwid-UM301 

to the Web server at URL www.training. This second form 
submission could lead to a continued session (e.g., a chance 
to register for another course). For this example, however, 
the remote service terminates the session with a confirma- 
tion message shown in Table 7, which is similar to the 
message in Table 1, as previously discussed. 

TABLE 7 

<PMb 

<AUDIO SRC-"lhanltt3u"/> 
<AUDIO SRC-"coafirm^u7> 
Peter Malaga 

<AUD10 SRC-"coufse^u'7> 
UM301: Advanced Unified Messaging 
cAUDIO SRC-'*when.au"/> 
Januaiy 26, 1999 from 3:00pm to 5:00pm 
<fPMU 



This results in the following presentation to the recipient: 

"Thanks for registering. This confirms that Peter Mataga 
is registered for course UM301: Advanced Unified 
Messaging, scheduled to take place Jan. 26, 1999 from 
3:00 pm to 5:00 pm." 

"Message processing complete." 

(The messaging system, them proceeds to offer the recipi- 
ent the usual mailbox actions such as delete, next, etc.) 

The advantage of this above-discussed mechanism is that 
the original structured message, which may possibly be a 
bulk email sent to many recipients, is small. When some 
subset of recipients choose to respond, the messaging sys- 
tem 104 again performs much of the processing associated 
with gathering information. 

The flowchart in FIG. 7 illustrates the steps associated 
with sending to a recipient a structured message in which, 
during a recipient's interaction with that structured message, 
information, such as a second structured message, is 
retrieved from a destination specified within the message, to 
possibly enable further interaction between the recipient and 
the information retrieved from that destination. Steps 701, 
702 and 703 are identical to steps 201, 202 and 203, 
respectively, described above in connection with the flow- 
chart in FIG. 2, and, for brevity, arc not repeated here. At 
step 704, the messaging system interprets the structure of 
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message from embedded instructions and plays to the recipi- 
ent a message having at least one embedded link to a 
destination where a second structured message is stored. 
During playing of the first message, an input from the 

5 recipient is requested to select a particular link. At step 70S, 
the recipient selects a link to that destination. At step 706, 
the messaging system retrieves the second structured mes- 
sage from the linked destination. At step 707, the messaging 

J0 system interprets the structure of the retrieved second struc- 
tured message from its embedded instructions and plays to 
the recipient the second message, which may request input 
from the recipient at specific points during play-out of that 
message. At step 708, the recipient supplies to the messaging 

is system the requested inputs at those specific points during 
play-out of the second message. At step 709, the messaging 
system forwards to a destination specified within the second 
message, the information collected during the play-out of 
the second structured message. The response to that can be 
another structured message (not shown), or, al step 710, the 
continuation of the recipient's mailbox session where the 
recipient can delete or replay the original message, access 
another stored message, or end his mailbox session. 

25 The various substructure types described above can be 
combined in various ways. For example, a structured mes- 
sage (or sequence of such messages) can cause coordinated 
data and telephony actions. In this example, the messaging 
system 104 collects inputs from the recipient at telephone set 

30 106, communicates data to a server, and then also places a 
telephone call over the PSTN to a phone number associated 
with the data destination. On call answer, DTMF signaling 
(or another signaling method) is used to transmit sufficient 

35 information (e.g., a reference ID) to allow the system 
receiving the telephone call to access the form data that was 
separately sent. The receiving system may then use that data 
to enhance the handling of the telephone call in various ways 
such as routing it to an appropriate agent, and/or providing 

40 a screen pop of the submitted form data and retrieved data 
to the agent's terminal. Alternatively, an IP telephony call 
could be placed to an IP telephony number associated with 
the data destination. 

45 As an example, it is postulated that the registration 
process requires interaction with a human agent, but that it 
is desired to collect data from the registrant automatically 
rather than through the live interaction. The messaging 
system 104 can be used to collect the information, essen- 

50 tially becoming pan of a distributed call center. This infor- 
mation can be collected before any phone call is made to the 
human registration agent, thereby potentially representing a 
substantial reduction in the load on the phone lines to the 

55 registration department's call center as compared to what 
would otherwise be a traditional call center which would 
need to maintain a dedicated voice response system. 

One way to achieve coordination between the form sub- 
mission and the outcall can be demonstrated in the previous 

60 example, if the response to the form submission containing 
the course registration information is not a simple confir- 
mation message, but a response indicating that a call should 
be placed. The information for the call preferably will 
indicate a session ID or some other identifier, which the 

65 system receiving the call interprets in order to retrieve the 
associated data. Table 8 illustrates an example of a respon- 
sive page to the registration. 
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<PML> 

cAUDIO SRC--thaiil£s.au'7> 
<AUDIO SRC«"oonfinn.nu"/> 
Peter Malaga 

<AUDIO SRC»"coui5e.au7> 

UM301: Advanced Unified Messaging 

<AUDIO SRC-"when.au"/> 

January 26, 1999 from 3:00pm to 5:00pm. 

<USK TYPE^ALL DEST--pstn:630-555-4444" DTMF-"3,2,9S765" IMMEDIATE> 

<AUDIO SRCVagcnUau7> 
<fUWK> 
</PML> 



This deviates from the previous example in which, after 
registering, only the confirmation message of Tabic 7 is 
played to the recipient. In this case, the call proceeds without 
the recipient's initiation because of the IMMEDIATE 
attribute of the LINK, The following is therefore the inter- 2Q 
action: 

"Thanks for registering. This confirms that Peter Mataga 
is registered for course UM301: Advanced Unified 
Messaging, scheduled to take place Jan. 26, 1999 from 
3:00 pm to 5:00 pm." ^ 

"A call is being placed to a customer agent at phone 
number 630-555-4444. Press pound to terminate the 
call." 

(The messaging system dials 630-555-4444 [the phone 
number associated with the registration system call 30 
center 125 in FIG. 1], waits for answer, sends DTMF 
tone 3, pauses, then sends tone 2, pauses, then sends 
tones 98765, conferences in voicemail user, then wails 
for call termination) 
(The session ID 98765 is used by the receiving system to 35 
route the call to an appropriate agent at telephone 126, 
and to trigger a screen pop of the submitted registration 
information on the agent's client terminal 127.) 
In above mechanism, the called service is assumed to 
have the special functionality that enables the interaction to to 
take place. Since the service owner is presumably the sender 
of the message, the service owner would be so configured. 
Advantageously, as in previous examples, the preparation of 
the registration data is completed before the establishment of 
the phone call, thereby reducing the load on the call center. 45 

The flowchart in FIG. 8 illustrates the steps associated 
with sending to a recipient a structured message in which, 
during a recipient's interaction with that structured message, 
information is collected from the recipient and sent to a 
specified destination, and a separate call (telephone over 50 
PSTN, IP telephony over data network, or any other type of 
call) is established by the messaging system to a number 
associated with that specified destination. Steps 801, 802 
and 803 are identical to steps 201, 202 and 203, respectively, 
described above in connection with the flowchart in FIG. 2, 55 
and, for brevity, are not repeated here. At step 804, the 
messaging system interprets the structure of the message 
from the embedded instructions and plays the message to the 
recipient, requesting input from the recipient al various 
points during play-oul. At step 805, the recipient supplies 60 
audio and/or keypad input at the requested points during 
message play-out. At step 806, if the recipient provides 
audio input, ASR circuitry in the messaging system converts 
the audio input to textual or numerical information, as 
appropriate. At step 807, the messaging system sends the 65 
information collected from the recipient to a destination 
specified within the structured message. At step 808, in 



parallel, at the same time or later after receiving another 
structured message from the destination, a separate call 
(such as a telephone call over the PSTN or an IP telephony 
call over data network) is placed to the destination, or to a 
second destination from which the collected information can 
be retrieved. At step 809, the messaging system provides 
information to that first or second destination that identifies 
in some manner the collected information. At step 810, the 
recipient interacts over the call with an agent at the first or 
second destination who has available the collected informa- 
tion. If the call to the first or second destination has not 
effected a transfer of the recipient to such destination, al step 
811, after completing the call, the recipient may continue 
with his mailbox session. 

With reference now to FIG. 9, a block diagram of inte- 
grated messaging system 104 is shown. The messaging 
system 104 includes sysiem software 901 which provides 
the operating system and associated software for hardware 
interfaces to enable communication with individual hard- 
ware components. Further, the system software performs 
those conventional messaging functions such as controlling 
the receipt of conventional audio, conventional email, and 
structured type of messages as described above, storage of 
such messages in a message storage database 902 for each 
recipient, deletion from storage of such messages in 
response to a recipient's direction, etc. Running on top of 
system software 901 is a structured message interpreter 
(SMI) 903 which, when a structured type of message is 
retrieved by a recipient from message storage 902, interprets 
the embedded instructions (the markup in the embodiment 
described above) and converts the structured message docu- 
ment into audio delivery for presentation to the recipient and 
with which the recipient may interact, if so structured, 
through audio means. Further, SMI 903 controls the playing, 
at the appropriate times as defined in the structured message, 
of audio files that are associated with the message and that 
may be attached to the message or retrieved from elsewhere. 
Functioning of system software 901 and SMI 903 is effected 
through a central processing unit 904 and associated RAM/ 
ROM memory 905. Various other modules are also included 
within messaging system 104, which can be implemented in 
hardware, software, and/or a combination of hardware and 
software. Further, some of the modules may effect process- 
ing within CPU 904 rather than within the module itself. 

A first network interface module is data network interface 
(DNI) 906, into which IP network 105 is terminated. DN1 
906 thus receives delivery of a structured message sent over 
the network by a sender, which it then forwards to message 
storage 902 for later retrieval by the intended recipient. 
Further, as previously described, when information is col- 
lected from a recipient during retrieval of an interactive 
message, DNI 906 outputs that information for transmission 
to a destination connected to IP network 105 if so specified 
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within the structured message. A second network interface 
module is telephone network interface (TNT) 907, to which 
PSTN telephone lines, such as a tip/ring analog interfaces or 
Tl digital lines, arc terminated. A recipient's PSTN tele- 
phone call to messaging system 104 to retrieve his messages 
thus terminates at TNI 907. TNI 907 thus also outputs the 
play-out of a retrieved structured message in accordance 
with the message's embedded instructions, as interpreted by 
SMI 903. Further, the keypad or audio inputs entered by the 
recipient during play-out of the message as navigational 
selections or input of solicited information are inputted 
through TNI 907. TNI 907 and DN1 906 are both connected 
to a system bus 908, as are the other illustrative modules 
shown in FIG. 9: audio play/record (P/R) module 909; 
text-to-speech (TTS) module 910; automatic speech recog- 
nition (ASR) module 911;and dual-tone multi-frequency 
(DTMF) detector module 912. 

Audio P/R module 909 includes the hardware and/or 
software for playing the recorded audio that has been stored 
as a data file and attached to a message. That message can 
be a conventional email message with a .WAV attached file, 
or a structured message of the type discussed herein above, 
where recorded audio messaging elements or message 
fragments, as previously described, are attached as data files 



part of the message which, when received by the messaging 
system and retrieved by the recipient, will be converted from 
text into speech. The structured message is created by the 
sender by selecting from those predetermined textual frag- 

5 ments available at Tj-T^, audio fragments available through 
Aj-A^, and customized inputted audio fragments and tex- 
tual fragments, and dragging them, using a mouse, into the 
message box 1012, to create a desired message structure. 
The selected audio fragments, 1013, 1014 and 1015 are 

10 combined with selected textual fragments 1016, 1017 and 

1018, as intended by the sender and in a structured format 
selected by the sender. By selecting a "create a menu" button 

1019, the sender creates a navigational nest of options that 
will be presented to the eventual recipient, including, for 

15 example, the audio fragment 1013, and textual fragments 
1016 and 1017. Upon selecting the "create a menu" button 
1019, the sender may be queried as to what prompts are to 
be given to the recipient, what titles are to be associated with 
each selective choice, and what messaging elements are to 
10 be incorporated within the menu. Further, the message may 
include links 1020 and 1021, which can be selected by the 
sender from one of a predetermined number of recorded and 
stored links Lj-L^, or inputted by the sender through win- 
dow 1025. When the message has been formulated by the 



to the structured message or retrieved from a specified 25 sender, and a destination entered into destination window 



source. Audio P/R module 909 may also convert a recipi- 
ent's responsive audio input within a structured input, which 
may then be converted to a data file and outputted through 
DN1 906 as an attached file for transmission to a specified 
destination on IP network 105. 

Text-to-speech (TTS) module 910, which may be imple- 
mented in hardware, software, or a combination of hardware 
and software, includes a digital signal processor, which may 
be implemented within the module itself or on CPU 904. 
TTS module 910 converts the textual fragments within the 
structured message to speech during play-out of the message 
to the recipient. Automatic speech recognition (ASR) mod- 
ule 911, which may also be implemented in hardware, 
software, or a combination of hardware and software, moni- 



1026, the sender selects the send button 1027, and the editor 
generates a PML-formatted version of Ihe structured 
message, with attached data representations of the included 
audio files, which is sent to the recipient at the indicated 
30 destination. 

The structured message could also be created "by hand" 
with a text editor and an audio file recording utility. 

Although described in terms of audio messaging, it should 
be apparent to one skilled in the art, that the present 
35 invention could be equally be applied to multimedia 
messaging, in which a structured multimedia message would 
include a plurality of messaging elements, which may 
include video fragments, which be assembled by the mes- 
saging system in accordance with embedded instructions 



tors the recipient's audio input, recognizing the recipient's to included within the structured message. Thus, a structured 



utterances. These recipient inputs are converted into com- 
mands or recognized as inputs, depending upon where 
during the message play-oul they are received. Dual-lone 
multi-frequency (DTMF) module 912, which also can be 
implemented in hardware, software, or a combination of 45 
hardware and software, recognizes recipient's keypad 
entries. These keypad entries may also be commands or 
inputs, depending upon the message structure where, for 
example, the recipient may make a keypad entry during 
message play-oul to select a link, or the recipient may make 
a keypad entry as an input as part of an information 
collection process. 

As previously described, a structured message can be 
created by a sender through an editor running on a client 
terminal. An example of an editor's GUI 103 that can be 
used by a sender to create a structured message is shown in 
FIG. 10. As can be noted, GUI 103 includes a plurality of 
predefined textual and prerecorded audio fragments that are 
stored and accessed through buttons T^-Ty and Aj-A^, 
respectively. The sender has these predefined fragments 
available to formulate a message. By activating an audio 
recorder through button 1010, the sender can record, using 
a microphone associated with the client terminal, one or 
more audio clips that can be incorporated into the message 
and affixed to Ihe message as audio data files. Similarly, by 
inputting text through the text input window 1011, the 
sender can create textual fragments to be incorporated as 



multimedia message could include also provide the recipient 
with the navigational options described above in connection 
with the audio message, as well as the ability to link to 
another destination, as well as the other previously described 
aspects of structured messaging for audio messaging. 

Further, although the above-described embodiment incor- 
porated a phone markup language as the methodology of 
embedding instructions that define the structure of the 
message and which are used by the interpreter at the 
messaging system to play-out the message and offer Ihe 
recipient the navigational choices within the message, there 
are other ways that such instructions could be embedded 
within the message that do not use a markup language. Thus, 
for example, the message could be formulated using an 
Electronic Data Interchange (EDI), or any other kind of 
pre-agreed interchange format between senders and the 
interpreter running on the messaging system, that allow the 
messaging system to interpret the message structure and 
present to the recipient the structured message in accordance 
60 with those instructions. 

The foregoing merely illustrates the principles of the 
invention. It will thus be appreciated that those skilled in the 
art will be able to devise various arrangements which, 
although not explicitly described or shown herein, embody 
the principles of the invention and are included within its 
spirit and scope. Furthermore, all examples and conditional 
language that have been recited herein are principally 
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intended expressly to be only for pedagogical purposes to 
aid the reader in understanding the principles of the inven- 
tion and the concepts contributed by the inventors to fur- 
thering the art, and are to be construed as being without 
limitation to such specifically recited examples and condi- 5 
tions. Moreover, all statements herein reciting principles, 
aspects, and embodiments of the invention, as well as 
specific examples thereof, are intended to encompass both 
structural and functional equivalents thereof. Additionally, it 
is intended that such equivalents include both currently 10 
known equivalents as well as equivalents developed in the 
future, i.e., any elements developed that perform the same 
function, regardless of structure. 

Thus, for example, it will be appreciated by those skilled 
in the art that the block diagrams herein represent conceptual ]5 
views of illustrative circuitry embodying the principles of 
the invention. Similarly, it will be appreciated that any flow 
charts, flow diagrams, pseudocode and the like represent 
various processes which may be substantially represented in 
computer readable medium and so executed by a computer 20 
or processor, whether or not such computer or processor is 
explicitly shown. 

The functions of the various elements shown in the FIGs., 
including functional blocks labeled as "processors" or "serv- 
ers" may be provided through the use of dedicated hardware js 
as well as hardware capable of executing software in asso- 
ciation with appropriate software. When provided by a 
processor, server or computer, the functions may be pro- 
vided by a single dedicated processor, by a single shared 
processor, or by a plurality of individual processors, some of 30 
which may be shared. Moreover, explicit use of the term 
"processor", "server", or "computer" should not be con- 
strued to refer exclusively to hardware capable of executing 
software, and may implicitly include, without limitation, 
digital signal processor (DSP) hardware, read-only memory 35 
(ROM) for storing software, random access memory 
(RAM), and non-volatile storage. Other hardware, conven- 
tional and/or custom, may also be included. 

In the claims hereof any element expressed as a means for 
performing a specified function is intended to encompass i0 
any way of performing that function including, for example, 
a) a combination of circuit elements which performs that 
function or b) software in any form, including, therefore, 
firmware, microcode or the like, combined with appropriate 
circuitry for executing that software to perform the function. 43 
The invention as defined by such claims resides in the fact 
that the functionalities provided by the various recited 
means are combined and brought together in the manner 
which the claims call for. Applicant thus regards any means 
which can provide these functionalities as being equivalent 50 
to those shown herein. 
The invention claimed is; 
1. An electronic message comprising: 
a plurality of messaging elements, at least some of the 
messaging elements being content-related that are each ss 
associated with a portion of the content of the message 
and at least one of the messaging elements comprising 
instructions that define a structure of the message from 
which at least some of the content-related messaging 
elements can be sequentially combined for presentation 60 
to a recipient as a unified message; and 
an address of the recipient of the message on a messaging 
system that stores the message and is capable of 
interpreting the instructions, assembling and combin- 
ing at least some of the content-related messaging 65 
elements in accordance with the instructions, and 
sequentially presenting the assembled and combined 



774 Bl 

30 

content-related messaging elements to the recipient as 
a unified message when the recipient retrieves the 
message from storage. 

2. The electronic message of claim 1 wherein the message 
is a voicemail message. 

3. The electronic message of claim 1 wherein the message 
is a multimedia message. 

4. The electronic message of claim 2 wherein the content- 
related messaging elements comprise at least one from a 
group including: textual fragments, speech fragments, and 
references to speech and/or textual fragments stored at 
specified locations. 

5. The electronic message of claim 4 wherein the speech 
fragments comprise references to audio data files associated 
with the message. 

6. The electronic message of claim 4 wherein the instruc- 
tions are embedded in the message using a phone markup 
language. 

7. The electronic message of claim 1 wherein the instruc- 
tions define a hierarchical menu structure which, when the 
message is retrieved by the recipient and at least some of the 
content-related messaging elements are assembled and com- 
bined by the messaging system and sequentially presented to 
the recipient, offers a plurality of navigational choices within 
the message to the recipient, at least one previously non- 
presented content -related messaging element next presented 
to the recipient being determined by a selection by the 
recipient of at least one of the navigational choices. 

8. The electronic message of claim 1 wherein a link to a 
destination is associated with at least one of the content- 
related messaging elements presented to the recipient. 

9. The electronic message of claim 8 wherein the link 
defines an address for establishing a telephony connection. 

10. The electronic message of claim 9 wherein the address 
for establishing the telephony connection is a telephone 
number on the PSTN. 

11. The electronic message of claim 9 wherein the address 
for establishing the telephony connection is an IP telephony 
address. 

12. The electronic message of claim 8 wherein the link 
defines an address for delivering a message. 

13. The electronic message of claim 12 wherein the 
address for delivering a message is an email address. 

14. The electronic message of claim 12 wherein the 
address for delivering a message is a Web address. 

15. The electronic message of claim 8 wherein the link 
defines an address for retrieving a second electronic mes- 
sage. 

16. The electronic message of claim 15 wherein the 
second electronic message comprises a plurality of its own 
messaging elements, at least some of the messaging ele- 
ments in the second message being content-related that are 
each associated with a portion of the content of the second 
message, and at least one of the messaging elements in the 
second message comprising instructions that define a struc- 
ture of the second message from which at least some of the 
content-related messaging elements in the second message 
can be sequentially combined for presentation to the recipi- 
ent as a unified message. 

17. The electronic message of claim 1 wherein the instruc- 
tions define a message structure which, when the message is 
retrieved and at least some of the content-related messaging 
elements are assembled and combined and sequentially 
presented to the recipient, input of information from the 
recipient is requested and collected by the messaging system 
and sent to a destination indicated in the message. 

18. The electronic message of claim 17 wherein the 
information from the recipient includes recipient's keypad 
entries. 
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19. The electronic message of claim 17 wherein the 
information from the recipient includes recipient's audio 
input. 

20. An electronic message comprising: 
a plurality of message fragments, each associated with at 

least a portion of the content of the message; 

a plurality of instructions embedded within the message 
that together define a structure of the message, said 
instructions including information indicating how at 
least some of the message fragments arc to be sequen- 
tially combined for presentation to a recipient as a 
unified message; and 

an address of the recipient on a messaging system that 
stores the message and is capable of interpreting the 
instructions, assembling and combining at least some 
of the message fragments in accordance with the 
instructions, and sequentially presenting the assembled 
and combined message fragments to the recipient as a 
unified message when the recipient retrieves the mes- 
sage from storage. 

21. The electronic message of claim 20 wherein the 
message is a voiccmail message. 

22. The electronic message of claim 21 wherein the 
message fragments comprise at least one from a group ^ 
including: textual fragments, speech fragments, and refer- 
ences to speech and/or textual fragments stored at specified 
locations. 

23. The electronic message of claim 22 wherein the 
speech fragments comprise references to audio data files. 

24. The electronic message of claim 22 wherein the 
instructions arc embedded in the message using a phone 
markup language. 

25. The electronic message of claim 20 wherein the 
message is a multimedia message. 

26. The electronic message of claim 20 wherein the 
instructions define a hierarchical menu structure which, 
when the message is retrieved by the recipient and at least 
some of the message fragments are assembled and combined 
by the messaging system and sequentially presented to the 
recipient, offers a plurality of navigational choices within the 
stored message to the recipient, at least one previously 
non-presented message fragment next presented to the 
recipient being determined by a selection by the recipient of 
at least one of the navigational choices. 
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27. The electronic message of claim 20 wherein a link to 
a destination is associated with at least one of the message 
fragments presented to the recipient. 

28. The electronic message of claim 27 wherein the link 
defines an address for messaging. 

29. The electronic message of claim 28 wherein the 
address for messaging is an email address. 

30. The electronic message of claim 28 wherein the 
address for messaging is a Web address. 

31. The electronic message of claim 27 wherein the link 
defines an address for a telephony connection. 

32. The electronic message of claim 31 wherein the 
address for the telephony connection is a telephone number 
on the PSTN. 

33. The electronic message of claim 31 wherein the 
address for the telephony connection is an IP telephony 
address. 

34. The electronic message of claim 27 wherein the link 
defines an address for retrieving a second electronic mes- 
sage. 

35. The electronic message of claim 34, wherein the 
second electronic message comprises a plurality of its own 
message fragments, each associated with at least a portion of 
the content of the second message; and a plurality of 
instructions embedded within the second message that 
together define a structure of the second message, said 
instructions for the second message including information 
indicating how at least some of the message fragments in the 
second message are to be sequentially combined for pre- 
sentation to the recipient as a unified message. 

36. The electronic message of claim 20 wherein the 
instructions define a structure which, when the message is 
retrieved and at least some of the message fragments are 
combined and sequentially presented to the recipient, input 
of information from the recipient is requested and collected 
by the messaging system and sent to a destination indicated 
in the message. 

37. The electronic message of claim 36 wherein the 
information from the recipient includes recipient's keypad 
entries. 

38. The electronic message of claim 36 wherein the 
information from the recipient includes recipient's audio 
input. 
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